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ABSTRACT 

This report outlines the work and results of an Educational 
Assessment Task Force at Mount Hood Community College (Oregon) that was 
charged with improving institutional planning, strengthening programs and 
services, and meeting external accountability demands. The task force 
addressed both institutional effectiveness (the annual assessment of progress 
in achieving institutional goals using a series of key indicators) and 
educational assessment (a similar process but with a focus on instructional 
programs, disciplines and course offerings). To assess institutional 
effectiveness, the task force adopted Nichols’ Five-Column Model, which 
addresses: (1) institutional goals; (2) intended educational outcomes; (3) 

means of program assessment and criteria for success; (4) summary of data 
collected; and (5) use of results. To implement the Educational Assessment 
Design, the task force utilized a peer-led model with faculty from well- 
assessed programs mentoring other faculty through a pilot process. The 
assessment timeline is presented along with a description of the roles and 
responsibilities of the assessment oversight team. Appended are: key 
assessment indicators for each institutional goal; data table examples; and 
"Mt. Hood Community College Educational Assessment Pilot Test Findings and 
Recommendations (prepared for Mt . Hood Community College's R. Dan Walleri by 
RMC Research Corporation's Gwen Hyatt, June 2001). (RC) 
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INSTITUTIONAL EFFECTIVENESS AND EDUCATIONAL 

ASSESSMENT 
Final Report 

EXECUTIVE SUMMARY 

Institutional Effectiveness (IE) refers to the annual assessment of progress in achieving 
institutional goals. The assessment is conducted using a series of indicators that measure specific 
aspects of the college goals. Educational Assessment (EA) is a similar process but with a focus on 
instructional programs, disciplines and course offerings, typically college units (Nursing, History, 
Continuing Education, etc.). IE and EA should be logically connected. For example, under the 
college goal, Knowledge-Based Workforce, one of several indicators is graduation/completion. At 
the level of IE, one would review the overall graduation rate for all students. At the EA level, one 
would be examining the graduation rate for specific programs, such as Nursing. All IE indicators 
should be relevant to all or a subset of units, such as professional technical programs or transfer 
disciplines. Not all EA indicators need to relate back to IE indicators. For example, licensing 
examination results only apply to selected programs, such as Nursing. Finally, IE and EA 
indicators should have standards or benchmarks against which institutional and unit effectiveness 
can be determined. An example is the longstanding statewide standard of 70% job placement rate 
for graduates of professional technical programs. 

This report describes the work and results of the Educational Assessment Task Force. Initiated in 
the spring of 1999 by Vice President of Instructional Services and Community Development, 

Mike Durrer, the Task Force has accomplished the following through spring 2001. 

♦ Reviewed regional accreditation standards 

♦ Crafted a statement of purpose 

♦ Designed a system for assessing IE and instructional units 

♦ Prepared initial college report on IE (fall 1999, subsequently updated annually) 

♦ Initiated pilot EA efforts 

♦ Developed schedule for EA of all instructional units 

♦ Evaluated and refined EA 

♦ Establishment of permanent committee to oversee IE and EA at MHCC — the 
Educational Assessment Oversight Group, including community representation 

By the end of the 2001-2002 year, the IE and EA system for MHCC should be fully implemented. 
If successful, this effort should assist in improving institutional planning and performance, 
strengthen programs and services, and meet external accountability demands, such as regional 
accreditation. 
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Educational Assessment . . . 

Statement of Purpose 

As an educational institution, MHCC “has an obligation to plan its courses of instruction to 
respond to student needs, to evaluate the effectiveness of that educational program in terms of the 
change it brings about in students, and to make improvements in the program dictated by the 
evaluative process” (Northwest Association of Schools and Colleges, Accreditation Handbook, 
1999 Edition, pp. 36-37). The purpose of MHCC’s educational assessment effort is to: 

• Facilitate student success 

• Ensure that students are receiving effective instruction and achieving the 
standards set by programs, disciplines and course offerings 

• Improve programs, disciplines and course offerings including assessment of 
needs related to staff, curriculum, equipment, facilities and other resources 

• Provide the administration, board and other appropriate bodies with 
assessment findings to be used in decisions about the initiation, continuation, 
enhancement and/or elimination of programs, disciplines and course 
offerings 

• Respond to external accountability demands, including accreditation 
The educational assessment process at MHCC will be characterized as: 

• Collaborative (with participation and support from the board, administration, 
faculty, support staff, students and community) 

• Ongoing (a dynamic process that is expected to evolve over time - 
continuous improvement) 

— — • Impartial — — 

• Efficient 

• Data driven 

• Meaningful (in that the integrity of the process will be preserved and 

honored) 

• Accountable (those involved will hold each other accountable for following 
the process, making decisions and following through) 

• Communicated (assessment results will be shared before any decisions, and 
outcomes will be communicated to stakeholders) 
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I. Introduction and Background 

Increasing demands for accountability has been one of the defining characteristics of 
postsecondary education in the 1990s. Whether at the federal level (Student Right-to-Know Act), 
state level (Oregon State Board of Education “Performance Measures” for the community 
colleges), or locally (planning and budgeting, accreditation), assessing institutional effectiveness 
and educational offerings is an ongoing challenge. MHCC’s accreditation review, conducted in 
fall 1997, highlighted accountability through two general recommendations to “assess institutional 
effectiveness based on a new or refined list of Indicators of Effectiveness and periodically make 
the results public, and to “eliminate inconsistencies and unevenness in educational program 
assessments.” Educational assessment is also an important component of developing the 
institutional master plan, MHCC 2010, in terms of anticipating future curriculum and service 
needs for the community. However, the primary motivation behind the involvement of Task 
Force members is to enhance opportunities for student success and provide a basis for 
continuous improvement of programs and course offerings. 

The Educational Assessment Task Force, co-chaired by Vem Porter, Instructor in Welding, and 
Dan Walleri, Director of Research and Planning, has developed: 

□ Indicators of institutional effectiveness; 

□ Institutional effectiveness report (fall 1999, subsequently updated annually); 

□ Content and format for program/discipline assessment reports; 

□ Content and format for program improvement plans; and 

□ Has guided implementation of pilot program/discipline assessments. 

For additional information see Institutional Effectiveness and Educational Assessment Design 
Document (fall 1999), Institutional Effectiveness (IE) Report (fall 1999 and fall 2000), and 
Institutional Effectiveness and Educational Assessment Data Notebook (January 2000). The 
ongoing work of the Task Force is described in progress reports published in June 2000 and 
December 2000. Completing its work, the Task Force disbanded in June 2001. An Educational 
Assessment Oversight Group (OG), including community representation, has been established to 
monitor and evaluate assessment efforts on an ongoing basis. The OG began meeting in April 
2001. 

Institutional Effectiveness (IE) refers to the annual assessment of progress in achieving 
institutional goals. The assessment is conducted using a series of indicators that measure specific 
aspects of the college goals. Educational Assessment (EA) is a similar process but with a focus on 
instructional programs, disciplines and course offerings, typically college units (Nursing, History, 
Continuing Education, etc.). IE and EA should be logically connected. For example, under the 
college goal, Knowledge-Based Workforce, one of several indicators is graduation/completion. At 
the level of IE, one would review the overall graduation rate for all students. At the EA level, one 
would be examining the graduation rate for specific programs, such as Nursing. All IE indicators 
should be relevant to all or a subset of units, such as professional technical programs or transfer 
disciplines. Not all EA indicators need to relate back to IE indicators. For example, licensing 
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examination results are only applicable to selected programs. Finally, IE and EA indicators may 
have standards or benchmarks against which institutional and unit effectiveness can be 
determined. An example is the longstanding statewide standard of 70% job placement rate for 
graduates of professional technical programs. 

II. Institutional Effectiveness and Educational Assessment Design 

The college IE and EA effort is built around the current five college goals. 

Goal #1. Knowledge-Based Workforce. Provide knowledge-based educational offerings to meet 
student and community needs, with a complementary set of student and community support 
services. 

Action Strategies 

1.1 Strengthen instructional program with aim of achieving highest standards in preparation and 
outcomes. 

1 .2 Develop an outcomes-based integrated General Education program to compliment discipline 
specific preparation. 

1 .3 Develop exemplary instructional improvement and evaluation systems (faculty evaluation and 
educational outcomes assessment). 

1 .4 Design and implement activities to enhance the recruitment, hiring, orientation and mentorship of 
all 

college employees to support achievement of a knowledge-based learning community. 

1 .5 Strengthen business and industry partnerships with focus on the education and training needs of 
incumbent workers. 

Goal #2. Access & Diversity. Provide affordable and attractive option for members of the 
community seeking a post-secondary education and/or careers, including the creation of an 
environment in which diversity thrives. 

Action Strategies 

2.1 Increase annual FTE by 3%. 

2.2 Increase access throughout the district through the development of partnership-based community 
centers. 

2.3 Strengthen student recruitment efforts. 

2.4 Increase the number of local high school graduates attending MHCC. 

2.5 Strengthen developmental education preparation such that the achievement of under-prepared 
students will equal that of entering students not needing remediation. 

2.6 Conduct targeted outreach efforts and create an environment that provides support for an 
increasingly 

diverse student population to be successful. 

2.7 Increase the participation rate of local high school students in dual credit programs. 

2.8 Enhance employee sensitivity and appreciation of diversity. 

2.9 Improve employee recruitment to achieve a college workforce reflecting the diversity of the 
community. 

2.10 Develop Distance Education program to provide learning independent of time and place. 
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2.1 1 Develop and implement and interactive college Web site such that students can obtain 
comparable services via Web that they could in-person. 



Goal #3. Requirements of Economic Development. Develop programs in emerging technologies with 
emphasis on information, engineering, bio-medical and biological technologies. 

Action Strategies 

3.1 Design programming in high priority areas — information, engineering, bio-medical and 
biological technologies. 

3.2 Strengthen partnerships with local governments, workforce development agencies, and higher 
education. 

3.3 Strengthen programs to prepare a skilled workforce to support economic development in the 
region. 

3.4 Act as a catalyst to stimulate economic development. 

3.5 Design and implement a job development program. 

Goal # 4 . Transitions. Provide seamless transfer opportunities to colleges, universities and careers. 

Action Strategies 

4. 1 Strengthen university articulation and increase the number of MHCC students transferring to 
four-year colleges and universities. 

4.2 Strengthen career services and increase the job placement rate for MHCC graduates. 

4.3 Increase number of GED, ESL and other outreach program completers who successfully 
transition to college-level programs. 

4.4 Plan, design and build partnerships for establishment of a University Center at MHCC, allowing 
students and local residents the opportunity to earn a four-year degree. 



Goal #5. Student Success . Provide infrastructure and support services to ensure student success. 

Action Strategies 

5.1 Increase overall student retention rate by 3%. 

5.2 While preserving existing assets and facilities, plan and propose strategies for facilities 
expansion to meet the increasing demands of a community undergoing significant growth and 
cultural transformation. 

5.3 Provide sufficient state-of-the-art technology and equipment to support knowledge-based 
education (Information Technology Strategic Plan). 

5.4 Develop a comprehensive resource development program to ensure that financial requirements 
are secured. 

5.5 Enhance/increase student financial aid. 

5.6 Enhance childcare services for students. 

5.7 Enhance instructional support services (tutoring, mentorships, etc.). 

To assess IE, the Task Force created a “matrix of college goals and indicators of institutional 
effectiveness.” For each college goal, a series of indicators were established. For example, under 
the college goal, Knowledge-Based Workforce, indicators include graduation rate, jobs available 
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related to skill training, living wage, etc. For each indicator, a description and standard are 
provided both at the institutional and unit level. For example, the standard for job placement is 
70% both for the institution and all relevant programs. In other cases, the standard is relative, 
based on comparison to other colleges or other units within MHCC. 

The Task Force created a similar matrix to guide unit assessments. In addition to the IE 
information, this matrix describes to which programs an indicator applies (professional technical, 
transfer, service or all), description of indicators and measures, standards, and commentary. Some 
of the measurement data for indicators is supplied centrally from Research & Planning, while 
other measures are unique to the unit and derived from various sources. See Appendix A for 
complete listing of goals and indicators. 

Further information and details can be found in the IE Report (fall 1999 and fall 2000) and IE & 
EA Data Notebook (January 2000). Both are available from Research & Planning and are also on 
the web at http://www.mhcc.cc.or.us/ci/allabout/research/main.htm . 

III. Educational Assessment Pilot Teams 

To implement the EA design, the Task Force decided to use a peer-led model with faculty from 
well-assessed programs mentoring other faculty. The Allied Health programs undergo rigorous 
assessment for professional accreditation. Teri Tong, from Dental Hygiene, and George Hicks, 
from Respiratory Care, were recruited as initial mentors and worked with Chris Bruya and Mary 
Girsh, Communications & Performing & Visual Arts, Bill Wright, Business Administration, Vem 
Porter, Welding, and Jack Schommer, Community TV. The deans and faculty members will then 
work with their peers as the EA design is implemented across all instructional areas. 

The pilot teams have been working on articulating the goals and objectives of the particular 
program or discipline, defining appropriate indicators and measures, and identifying sources of 
assessment information. 

Gwen Hyatt of RMC Research Corporation completed an evaluation of pilot assessment efforts in 
the spring of 2001 (see Appendix C). 

IV. Educational Assessment Schedule 

Orientation and training for all faculty and deans began with the fall 2000 in-service and 
subsequent small group sessions. Divisions and programs actively engaged in educational 
assessment during 2000-2001 are as follows. 

□ Allied Health - all programs 

□ Business and Computer Technology - Business Administration 

□ Developmental Education - Adult Basic Education/GED 

□ Industrial - Welding 
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□ Communications, Performing and Visual Arts - Visual Arts and Graphic Arts Technology 

□ The Center for Community and Workforce Development - Continuing Education 

Remaining divisions and programs will be brought into the process during 2001-2002 beginning 
with additional training during the fall 2001 in-service. Educational assessment will subsequently 
be conducted on an annual basis. 

V. Educational Assessment Process at the Program/Discipline Level 

MHCC’s EA design has two basic components. The first is a review of indicators that would be 
done at the program or discipline level and would cover enrollment, cost, retention, job placement 
and so on (see Table 1). 



Table 1. Indicators Developed by the Assessment Taskforce 



□ 




Who?* 


Meaning 


Standard 


Data Source 


College Goal: Knowledge-Based Workforce 




Graduation/Completion 


PT 


Percentage of first-time 
degree-program students 
earning degree or 
completing core 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f< 
your discipline 




Jobs available related to skill training 


PT 


Determined by state and 
national projections, skill 
society research, advisory 
committee, local newspaper 
job announcements 


High (demand exceeds 
supply) or medium (balance 
in supply and demand) 


Assessment Data report f< 
your discipline 




Living wage 


PT 


Determined by State of 
Oregon 


$ 10/hour 


Assessment Data report f< 
your discipline 




Marketing 


All 


Marketing of specific 
programs 


Done as needed to maintain 
or enhance enrollment 


Self-assessment 


College Goal: Access and Diversity 




Enrollment 


All 


Number of students enrolled 
in program or with declared 
major in discipline 


Stable (less than 20% 
variance over 3-5 years) or 
growing (more than 20% 
increase) 


Assessment Data report f< 
your discipline 




Cost/Revenue per FTE 


All 


Derived from expenditures 
and budget 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f< 
your discipline 




Unique to Portland area 


PT 


Determined by Instructional 
Council 


Unique programs merit 
special consideration 


Self-assessment 
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0 




Who?* 


Meaning 


Standard 


Data Source 


College Goal: Transitions 




Job placement of graduates 


PT 


Percentage employed or 
continuing their education in 
field of training 


70% 


Assessment Data report f< 
your discipline 




Job placement of nongraduate core 
course completers 


PT 


Percentage employed or 
continuing their education in 
field of training 


60% 


Assessment Data report f< 
your discipline 




Number of transfers 


Trans/PT 


Total number of transfers per 
year 


Relative to other programs 
(top, middle, or bottom third) 


By request from Research 
and Planning 




Transfer performance 


Trans/PT 


Percentage maintaining 
satisfactory academic 
standing 


Relative to other programs 
(top, middle, or bottom third) 


By request from Research 
and Planning 


College Goal: Student Success 




Course success 


All 


Percentage of students 
completed selected courses 
with C or better 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f< 
your discipline 




Sequential course success 


Trans/ 

Other 


Percentage of students 
completing identified 
sequential courses with C or 
better 


Relative to other programs 
(top, middle, or bottom third) 


By request from Research 
and Planning 




Student satisfaction— Course 


All 


Percentage of students 
expressing satisfaction 


Relative to other programs 
(top, middle, or bottom third) 


Samples available from 
Research and Planning 




Student satisfaction— Exit 


PT/Trans 


Percentage of students 
expressing satisfaction 


Relative to other programs 
(top, middle, or bottom third) 


Samples available from 
Research and Planning 




Student satisfaction— Follow up 


PT/Trans 


Percentage of students 
expressing satisfaction 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f< 
your discipline 




Student satisfaction— General ed 
assessment 


Trans 


Percentage of students 
expressing satisfaction 


Relative to other programs 
(top, middle, or bottom third) 


Samples available from 
Research and Planning 




Faculty and program satisfaction with 
college support services 


All 


Satisfaction with support 
services, e.g., marketing, 
advising and counseling, 
business services) 


Meet educational needs of 
the instructional area 


Self-assessment 




Student, faculty, and program 
satisfaction with service received 


Other 


Instruction in services 
courses 


Meets the specific needs of 
other programs and 
disciplines 


Self-assessment 




Retention— Course 


All 


Percentage of students 
completing individual 
courses 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f( 
your discipline 
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0 




Who?* 


Meaning 


Standard 


Data Source 




Retention— Program 


PT 


Percentage of first-time 
degree-program students 
retained over 2 years 


Relative to other programs 
(top, middle, or bottom third) 


Assessment Data report f( 
your discipline 




Retention— Sequential courses 


Trans/ 

Other 


Percentage of students 
completing selected course 
sequences 


Relative to other programs 
(top, middle, or bottom third) 


By request from Research 
and Planning 




Technology funding level 


All 


Supply and operating 
budgets 


Adequate to meet the 
educational needs of the 
instructional area 


Self-assessment 




Equipment objective 


All 


Instructional equipment 


Adequate to meet the 
educational needs compared 
to industry standards 


Self-assessment 




Facilities funding level 


All 


Facilities funding 


Adequate to meet the 
educational needs of the 
instructional area 


Self-assessment 




Maintenance schedule 


All 


Facilities maintenance 


Adequate to meet the 
educational needs of the 
instructional area 


Self-assessment 


Supplemental Indicators (Program- or Discipline-Specific) 




Transfer performance relative to native 
students 


Trans 


Percentage maintaining 
satisfactory academic 
standing 


Relative to native students 


By request from Research 
and Planning 




Licensure 


PT 


Pass rate on program 
licensure examinations 


90% of students taking 
exams 


Self-assessment 




Diversity 


All 


Enrollment rates for under- 
served populations 


Set by discipline 


Assessment Data report f< 
your discipline 




Full- to Part-time IFTE Ratio 


All 


Percent of class hours 
taught by full-time instructors 


At least 60% full-time 


Assessment Data report f< 
your discipline 




Matriculation from Preparatory (e.g., 
ABE, ESL) to Credit Courses 


Other 


Percent of preparatory 
students matriculating to 
credit courses 


Set by discipline 


By request from Research 
and Planning 




GED Success Rate 


Other 


Percent of students in GED 
preparation who obtain GED 


Set by discipline 


By request from Research 
and Planning 
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Thus, the first step is for the program faculty to review the information on these indicators for the 
program and determine if an improvement plan is needed. Most of the needed information is 
available centrally from Research & Planning, which has prepared a data table for each program 
and discipline. However, program faculty may want to supplement with information available 
from professional organizations and other sources. For example, if the job placement rate for a 
professional technical program were below 70%, the program faculty would develop a plan of 
improvement. This would include a review of labor market information to determine availability of 
jobs in the field (see Appendix B for sample data table and findings). The review could also 
include discussion with program advisory committee members and employers to determine if 
curriculum is meeting the needs of the labor market. In addition to this analysis, the improvement 
plan would include action strategies designed to improve performance (increase use of 
cooperative work experience, outreach to employers, etc.). Annual ongoing review would allow 
the program faculty to assess the improvement plan to ensure that objectives are being achieved. 

This first component of EA is fairly straightforward with the indicators already tied back to 
institutional mission and goals as described in the IE and EA design document. In addition, 
program faculty will also need to define their intended educational outcomes specific to their 
program and assess those outcomes. This part of the EA process is what the pilot teams have 
been focused on. The process is often described as the “five-column model” from James O. 
Nichols’, A Practitioner’s Handbook for Institutional Effectiveness and Student Outcomes 
Assessment Implementation (Agathon Press, 3 rd edition, 1995). This process is summarized in the 
following diagram. 

Figure 1. Mission, Institutional Goals and Intended Educational Outcomes 
Institutional Level 




The steps in the five-column model can be summarized as follows. 

• Expanded Statement of Institutional Purpose (mission statement and institutional goals - 
completed at institutional level) 
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• Program Intended Educational Outcomes (defined at the program level, derived from 
expanded statement of institutional purpose) 

• Means of Program Assessment and Criteria for Success 

• Summary of Data Collected 

• Use of Results 

The table below illustrates use of the five-column model for different academic areas at MHCC 
(examples only - not based on actual assessments). 
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Table 2. Five-Column Model of Assessment (NOT ACTUAL, FOR ILLUSTRATION 
ONLY) 



Program 


College Goal 


Intended 

Outcomes 


Means of 

Assessment/ 

Criteria 


Summary of 
Findings 


Use of Results 


Dental Hygiene 


Teaching & 
Learning 


Students 
prepared for 
licensure/ 
employment 


90% of 
students taking 
licensure exam 
pass 


85% pass rate 


Revise specific 
courses and 
standards 
related to areas 
of licensing 
exam that 
students have 
difficulty with 


Dev Ed - Dev 
Writing 


Teaching & 
Learning 


Students 
prepared for 
college level 
writing courses 


Successful Dev 
Ed students 
perform equal 
to students not 
having to take 
Dev Ed 


60% of Dev Ed 
students 
successful in 
college writing 
compared to 
75% of non- 
Dev Ed 
students 


Revise and 
increase exit 
criteria from 
Dev Ed writing 
course 


Business Adm 


Teaching & 
Learning 


Students 
prepared for 
four-year 
college work 


Business Adm 
transfers 
perform equal 
to or better 
than other 
students in 
PSU Sch of 
Business 


MHCC 
Business Adm 
transfer 
students 
perform and 
graduate at 
higher rate than 
other transfers 
or native PSU 
students 


No 

improvement 
plan needed 



Demonstrating and documenting use of results is critical for “closing the loop” and feeding 
assessment results into the college’s planning process. It is also an increasing area of focus in 
regional accreditation review. 

VI. Oversight Group Operational Guidelines 



The “Oversight Group” (OG) will provide ongoing monitoring of the educational assessment 
process, results and actions. The OG will ensure that the criteria established in the Statement of 
Purpose for Educational Assessment are honored as follows. 

• Facilitate student success 



m:\data\word\assess\Final Report of the Ed Assess TF.doc 



15 



o 

ERIC 



16 



BEST COPY AVAILABLE 



MT. HOOD COMMUNITY COLLEGE 
Force 



Educational Assessment Task 



Research & Planning 

• Ensure that students are receiving effective instruction and achieving the 
standards set by programs, disciplines and course offerings 

• Improve programs, disciplines and course offerings including assessment of 
needs related to staff, curriculum, equipment, facilities and other resources 

• Provide the administration, board and other appropriate bodies with 
assessment findings to be used in decisions about the initiation, continuation, 
enhancement and/or elimination of programs, disciplines and course 
offerings 

• Respond to external accountability demands, including accreditation 

The group will publish a formal report on an annual basis. The President and President’s Council 
will utilize this report as input for improvement of programs, disciplines and course offerings, 
including meeting the needs related to staff, curriculum, equipment, facilities and other college 
related activities. Once reviewed by the President, the report will become part of the IE and EA 
presentation to the MHCCD Board. The OG will take special care in its monitoring and reporting 
role to ensure that the results of assessment and improvement plans receive appropriate 
administrative attention and follow up. 

Oversight Group Membership 

3 faculty (1 professional technical, 1 transfer, 1 Other, such as Developmental Ed.) 

2 administrators (2 deans) 

3 community members 
2 Support Staff 

1 ex-officio (Director, Research & Planning) 

Research and Planning will provide the support service and the Research and Planning director 
will serve as ex-officio to the OG. 

Terms of Office and Appointment 

Faculty: 2 years. Initial members will be staggered one and two years to place a new faculty on 
the group each year. Appointed by the Faculty Senate. 

Administrators: 2 years. Appointed by the College President. 

Community Members: 2 years. Initial members will be staggered one and two years to place a 
new community member on the group each year. Appointed by the College President. 

Support Staff: 2 years. Initial members will be staggered one and two years to place a new 
support staff member on the group each year. Appointed by the Classified Association. 

Student Member: 1 or 2 years as desired by the member. Appointed by the OG. 
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Members will serve no more that one term. The Chair for the group will be a faculty member. If 
vacancies occur, replacement shall be selected by a majority vote of the committee as a whole. 

Criteria for Membership 

1 . Experience/Expertise in Assessment (Faculty and Deans) 

(Be able to attend national Ed Assessment Workshop) 

2. Representation (Faculty/Deans/Support Staff) 

(A cross-section of professional-technical/transfer/other, Dev. Ed) 

3. Knowledge/Experience with College and Community (Faculty/Deans/Community/Support 
Staff) 

4. Time to do the Work (Faculty/Deans/Community/Support Staff) 

5. Excuse self from voting if directly involved in a program’s assessment 
(F aculty/Deans/Community/Support Staff) 



Meeting Frequency 

The OG will meet at least once each term of the regular academic year. Additional meetings may 
be scheduled as needed. 

Mission 

The “Oversight Group” has been formed for the purpose of monitoring and determining the 
effectiveness of the college in honoring the Statement of Purpose for Educational Assessment. 

The OG charge: 

• Review and evaluate college support related to the effectiveness of all course/program 
offerings 

• Review all assessment action plans and recommendations 

• Highlight and recommend ways to address reoccurring concerns documented in the 
assessment process 

• Compile feedback from faculty and staff on assessment process and make recommendations 
for needed revisions 

• Follow up on administrative actions in response to OG recommendations 

• Prepare an annual report to the faculty, administration and MHCC Board 

Prepare a formal assessment of the effectiveness of the OG every three years 
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Oversight Group Initial Meeting 
Current members of the OG are as follows. 

Table 3. Educational Assessment Oversight Group 2001-2002 



Community Members 
Ron Craig, Fujitsu 
Ed Hartin, City of Gresham 
Patty Brost, Legacy Mount Hood Medical Center 



Faculty 

Vem Porter, Industrial Division (chair) 

Rick Bolesta, Science Division 

Mary Girsch, Communication, Performing and Visual Arts Division 
Support Staff 

Rebecca Kenney, The Center for Community and Economic Development 
Garie Zordich, Language and Literature Division 

Administration 

Chris Bruya, Communication, Performing and Visual Arts Division 
John Saito, Allied Health Division 

Ex Officio 

Dan Walleri, Research and Planning 



The OG held its initial meeting on April 30, 2001 (see Appendix D for the minutes from this 
meeting). The members reviewed the assessment design, including the findings from an evaluation 
conducted by Gwen Hyatt of RMC Research Corporation. As a result of these discussions, 
further refinement of the design was made, with results incorporated into the overview of the 
educational assessment process as described in section VII. 

VII. Overview of Educational Assessment Process 

In addition to the design prepared by the Taskforce, the process described here also includes 
suggestions from Gwen Hyatt, outside evaluator from RMC Research, and from the Educational 
Assessment Oversight Group (OG). Both Hyatt and the OG recommended that the process be 
streamlined, especially with regard to the number of indicators that program faculty would be 
expected to respond to. 

Key Indicators . Each program should choose from among the supportive and customized 
indicators (see below) the 3 to 6 key indicators that most significantly measure the success of the 
program. Key indicators are those that are critical to the viability of a program (i.e., enrollment) 
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and will often be the same for groups of programs (professional technical, transfer, etc.). 

However, key indicators can be different for every assessment unit. For Continuing Education, 
one key indicator may be “Student Satisfaction — Course” because each individual continuing 
education course is taken by choice rather than by program design. For a transfer program, a key 
indicator may be success of MHCC students after they transfer compared to the success of other 
students at the transfer institutions. 

Supportive Indicators . Using the list of indicators developed by the Assessment Taskforce as a 
starting point, each program should respond to any indicators on the list that are relevant and 
meaningful. 

Customized Indicators . Each program should develop and respond to any other indicators it 
believes are good measures of its success. 

As conceived by the Assessment Taskforce, the function of the institutional assessment effort is to 
create an opportunity for feedback between the programs and departments and the college 
administration. The creation of the Assessment Oversight Committee is an important part of this 
process as it serves its function to “provide ongoing monitoring of the educational assessment 
process, results, and actions.” It is critical that there be a clear process in how the results and 
recommendations from the programs and departments will be transferred from the faculty and 
deans to the administration and how the administration will report back to the faculty. All 
recommendations resulting from this process that require resources or administrative approval 
need to follow a specific and consistent feedback loop as illustrated in figure 2. 

1 . The programs and disciplines make recommendations as part of the institutional 
assessment process. 

2. The deans submit overall division summaries to the President’s Council via the vice 
presidents and to the Director of Research and Planning who will work with the faculty 
chair to schedule review by the Assessment Oversight Committee. These division 
summaries report findings and recommendations from the programs and disciplines, and 
indicate which items need to be addressed in the next planning/budgeting cycle. Through 
the division summaries, the deans are responsible for bringing forward any resource 
requests identified in program and discipline assessments. 

3. As part of its annual planning and budget development process, the President’s Council 
responds in writing to the planning objectives and resource requests made by the deans as 
reported in the division summaries. The responses are sent to Assessment Oversight 
Committee and copied to the vice presidents, the deans, and the programs and disciplines. 

4. The Assessment Oversight Committee reviews the patterns of resource allocation and 
denial with special emphasis on the common indicators identified by each program or 
department. The committee submits an annual report to both the President’s Council and 
the MHCC District Board of Education. 

5. Based on findings from annual review of assessment indicators, the Oversight Committee 
may initiate a comprehensive review of a program or discipline on its own initiative or at 
the request of the President’s Council. A comprehensive review may take one of many 
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forms, such as additional data collection, paneling area employers, or DACUM. Results 
from a comprehensive review are forwarded to both the President’s Council and the 
Oversight Committee, with the Oversight Committee submitting any recommendations to 
the President’s Council. The President’s Council will issue its findings, recommendations 
and decisions in writing to the Assessment Oversight Committee and send copies to the 
vice presidents, the deans, and the programs and disciplines. The Assessment Oversight 
Committee will include a summary of results and recommendations in its annual report to 
the MHCCD Board of Education. 

Figure 2. The Assessment Reporting and Feedback Process 




VIII. Relationship to Planning and Resource Allocation 

Assessment, planning and budgeting are, or at least should be, ongoing processes. Time lines are 
only relevant because of legal, accountability and documentation requirements (for example, for 
accreditation). Thus, these various requirements require that snapshots be taken on a periodic 
basis. A complicating factor is that in the case of assessment you are looking back in time while 
for planning and budgeting you are looking ahead, at least one year and often two years. The 
following chart describes the time lines and relationship between assessment, planning and 
budgeting. The key time driver is the budget, which has to be adopted by the MHCCD Board of 
Education in June of each year for the fiscal year beginning July 1. Figure 3 provides an overview 
of the interrelated processes and time lines. 
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Figure 3. Assessment, Planning and Budget Time Lines 
ASSESSMENT PLANNING/BUDGETING 



March- April 



May 



Sept-Oct. 



January 



March 



June 
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I ppendix A. Indicators Per Criteria (Goal) 

Goal #1. Knowledge-Based Workforce. Provide knowledge-based educational offerings to meet student and community needs, 
with a complementary set of student and community support services. 

Key Indicator 

1.1 Graduation/Completion. Maintain level of graduates as a percentage of total annual 

FTE equal to or greater than that of comparable community colleges in the State of Oregon. 

1 .2 95% of MHCC graduates will pass licensure/certification examination results. 

1 .3 Living W age. Minimum of $ 1 0 per hour or $400 per week. Weekly income of 
professional technical graduates employed after attending MHCC (1 st quarter 
following graduation). 

1 .4 Achieve General Education outcomes (TBD). 

1.5 Faculty Participation. 100% of faculty participate in at least one professional 
development activity per year. 

1 .6 Faculty Satisfaction. At least 70% of the faculty rate the performance of the TLC as 
excellent or very good. 

1 .7 Customer Satisfaction. 90% of employers/employees rate customized training 
services as excellent or very good. 

1.8 Marketing. Expenditures and measures of effort relative to enrollment levels 
and growth. Benchmark to be determined. 

Goal #2. Access and Diversity . Provide affordable and attractive option for members of the community seeking a post-secondary education, 
including the creation of an environment in which diversity thrives. 

Key Indicator 

2.1 Enrollment. Increase annual FTE by 3% for 2002-2003. 

2.2 Community Satisfaction. From The Nelson Report (2001) Survey Research 

Report: “Mt. Hood Community College enjoys an excellent reputation in the community, 
gamering an extremely high positive rating of 67% (excellent - 20%, pretty good - 
47%). 

2.3 Increase market share of local high school graduates by 10%. 

2.4 Performance of Guided Studies Students. Successful Guided Studies students will 
achieve retention and academic performance levels comparable to those of non- 
Guided Studies students. 

2.5 Jump Start Enrollment. Increases by 5% in 2002-2003. 

2.6 Employee Profile. Percentage of protected classes among MHCC employees 
approximates community profile. 

2.7 Student Body Profile. Percentage of protected classes among MHCC students 
approximates community profile. 

2.8 Performance. Students from protected classes will matriculate, maintain standards 

of academic progress and complete programs of study at similar rates to students in non- 
protected classes. 

2.9 Cost/Revenue. Comparable to peer colleges. 

2.10 Achieve interactive Web time lines (student registration, etc.). 

2.1 1 Distance Education program development. Ten new courses developed each year. 

2.12 Distance Education enrollment - increase by 1,000 by 2002-2003. 

Goal #3. Requirements of Economic Development. 

Develop programs with emphasis on information, engineering, bio-medical and biological technologies. 
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Key Indicator 

3.1 Meet Program development and enrollment targets 
(TBD). 

3.2 Meet economic development and job creation targets 
(TBD). 

Goal # 4 . Transitions. Provide seamless transfer opportunities to colleges, universities and careers. 

Key Indicator 

4.1 Job Placement (Graduates). 70% of professional technical graduates are employed 
or continuing their education in field of training. 

4.2 Job Placement (Non-Graduate Core Class Completers, 60/30 core credits 
completed for associate degree/certificate). 60% of professional technical leavers are 
employed or continuing their education in field of training. 

4.3 Number of Transfers. Maintain level of transfer majors attending Oregon 
University System institutions. 

4.4 Transfer Performance. 90% of MHCC transfer students to OUS system 

maintain a minimum of 2.00 GPA. 

4.5 70% of students completing GED, ESL and other outreach programs who desire to will successfully 
transition to college-credit programs. 

Goal #5. Student Success. Provide infrastructure and support services to ensure student success. 

Key Indicator 

5.1 Retention. Increase overall retention rate by 3%. 

5.2 Student Satisfaction. Maintain "excellent/good" overall level of satisfaction with 
the college in general. 

5.3 Course Success. A minimum of 75% of students will receive a C grade or higher. 

5.4 Student Satisfaction. Maintain "excellent/good" overall level of satisfaction with 
the college in general. 

5.5 Increase Library base budget by 10% a year. 

5.6 Implement Facilities Master Plan. 

5.7 Facilities maintenance schedule is implemented according to 
specified time lines. 

5.8 Facilities Management customer satisfaction — TBD. 

5.9 Fund Raising. 10% increase annually. 

5.10 Capital Campaign. Targets achieved. 

5.11 Technology Funding Level. Maintain budget target for technology funding 
(currently 2.25% of annual operating budget). 

5.12 Computer User Satisfaction. Maintain "excellent/good" overall level of satisfaction. 

5.13 Equipment Objective. Provide sufficient state of the art equipment (monitored by 
funding distribution across areas). 

5.14 Campus Climate. Benchmark for campus climate established in survey conducted 
in spring 2000 with plans to repeat the survey each year. Indicator question: 
percentage agrees with statement that "MHCC is a supportive environment in which 
to work." 

5.15 Increase number of students receiving financial aid (target TBD). 

5.16 Increase number of childcare slots for students (target TBD). 

5.17 Increase number of students receiving instructional support services (target TBD). 



Research & 
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Finding: High on Course, Low on Program Retention 



MT. HOOD COMMUNITY COLLEGE Educational Assessment Task Force 



© 



WD 

a 

•»*N 

a 

a 

g 

S 

xs 

u 

Sm 

Cl 

© 

VS 

0) 



©| 

a 

«sO | C 
oH © 

x 

O 



^1 



©1 

a 

c 

(0 

X 

O 



o^l 



©I 

a 

c 

© 

a 



£1 & 

X 

O 



£1 



©i 

a 

c 

© 

X 

O 



O) 

c 



a> 



© 



D) 

C 

c 

it 



& £ 



© 

> 

© 



O) 

c 



© 

B 

X 

CJ 

© 

H 

I 

© 

*C 

© 

*c 

u 



■a 

c 

3 

Li. 

> 

O) 

o 

o 

c 

n 

u 

© 



O) 

c 

^5 

.c 

it 



c < 

© z 

E b> 

■■ I 

3 ^ 

o- .S 

LU U. 



© 

*x 

3 

© 

© 

fa 

i 

© 

■C 

© 

'C 

u 



© 

> 

© 



O) 

.E 

c 

3 



© 

© 



O 

© 



< 

.s? 

"8 

.st 

£ 



© 

3 

■a 

© 

.c 

u 

CO 

© 

u 

c 

© 

c 

© 

c 

’© 

s 



< 

z 

& 

"8 

£ 



LU 



© 

Q. 

CO 

o 

o 

_Q) 

-Q 

© 

C 

’© 
-*— > 
cd 
3 
cd 



© 

o 

3 

TU 

O 



O 

JZ 

£ 



© 

-4-^ 

© 

L. 

■O 

c 

© 

c 

© 

</> E 
5) o 
c c 
=5 ® 

c 2 

■ M © 

LL. © 

© 
3 
© 
CD 



2 

§>.o 

ol 



© 

'1 



cd 
© 

O CD 

> W 
© OT 
cd _c 

•g 3 
o CD 

% £2 
CD O 

© © 
0)0 
© “O 

=5.£ 

O <u 
**- CD 
O © 

c £ 

0 

~ O 
© m 

- p 
© t 

> o 
© 0 ) 

II 

1 2 

.2 fc. 

55 * 



o 

£ 

© 

> 

cd" 

© 

'■ 4 —' 

*c 

3 

-c 

o 

Q. 

Q. 

O 

_Q 

O 



© 

O) 

© 



O 3 t 



■Q 
3 



(D 55 

§ .1 

CD “O 
© "O 
*“ © 
© 

© 
o 



<S2 



■o 
© 
_> 
o 
> 
c 

© 
_Q 

,!= O 



■o 

© 

© 



c 
© 

E x> 
O) c 



© 



3 
O" 

2 © 
“O 

5 © 

2 c 

3 3 



"O 

© 

© 



© 

>> 

o 

CL 

E 

© 

© 



U) 

© 



c 

© 

o 



C _Q 






</> 

c 

o 

'■si 

"a 5 
C w 
a> c 

|? 

E < 

o 

O -r- 

0) 

QC 



“O 
© 
© 
© 
© 

« w 
© CD 
3 

© 

© _Q 
© C 

rti © 

s ° 

® c 
£ o 

o o 

II 



c 

O) 

CD o 
Q) © 



© 3 © ^ Q, 
**-rr CZ •— 
© 

E 

O "2 



C © 

* E 
° J 5 

si 

S3 



SO 

(N 



O 

co 



CD 

CM 




Mt. Hood Community College 
Educational Assessment Pilot Test 



Findings and Recommendations 



Prepared for 

Mt Hood Community College 

26000 SE Stark Street 
Gresham, OR 97030 



Prepared by 

RMC Research Corporation 

522 SW Fifth Avenue, Suite 1407 
Portland, OR 97204 



June 2001 




31 



Mt. Hood Community College Educational Assessment Pilot Test 



Findings and Recommendations 



Prepared for 

R. Dan Waller i 



Mt. Hood Community College 

26000 SE Stark Street 
Gresham, OR 97030 



Prepared by 



Gwen Hyatt 

RMC Research Corporation 

522 SW Fifth Avenue, Suite 1407 
Portland, OR 97204 



Contents 



Introduction 1 

The Educational Assessment Process 2 

Recommendations for Implementation 3 

Recommendation 1 : Customize the indicators and process to be used to each program .... 4 

Recommendation 2: Provide assistance and support for initial implementation 7 

Recommendation 3: Provide institutional support for the Unit Coordinators 10 

Recommendation 4: Provide ongoing support and coordination of the process 

through the Research and Planning department 10 

Recommendation 5: Close the feedback loop 11 

The Assessment Builder Software 13 

Problems With the Assessment Builder Interface 13 

Navigation 14 

Data Input 15 

Reporting 15 

Technical Problems With the Software 17 

General Recommendations 18 



Introduction 

In spring 1999 an Educational Assessment Taskforce was formed to meet Mt. Hood Community 
College’s (MHCC) need to address issues of institutional effectiveness. As explained in the 
MHCC Institutional Effectiveness and Educational Assessment Data Notebook: 

Increasing demand for accountability has been one of the defining characteristics of 
postsecondary education in the 1990s . . . MHCC’s accreditation review, 
conducted in fall 1 997, highlighted accountability through two general 
recommendations to “assess institutional effectiveness based on a new or refined 
list of Indicators of Effectiveness and periodically make the results public,” and to 
“eliminate inconsistencies and unevenness in educational program assessments.” 
Educational assessment is also an important component of developing the 
institutional master plan in terms of anticipating future curriculum and service 
needs for the community. 

The taskforce, co-chaired by Vem Porter, Welding Instructor, and Dan Walleri, Director of 
Research and Planning, worked to develop and guide the implementation of a collegewide 
program-level assessment process. This implementation began with a pilot test of faculty in 
selected divisions, including Allied Health; Business and Computer Technology (BCT); 
Communication, Performing, and Visual Arts; Industrial Technology; Continuing Education; 
Adult Basic Education; and Evening and Weekend College. 

The pilot test was carried out during the 2000-01 academic year. In April and May 2001, a 
consultant from RMC Research Corporation conducted interviews with 12 faculty members and 
deans representing transfer programs, professional/technical programs, and other programs that 
conducted the pilot test. The individuals interviewed are listed in Exhibit 1 . 
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Exhibit 1 

Pilot Test Participants Interviewed 



Name 


Department 


Program Type 


Ann Bonner 


Mental Health/Human Services 


Professional/Technical and Transfer 


Chris Bruya 


Art 


T ransfer 


Brenda Button 


ABE/GED 


Other 


Mary Girsch 


Art 


Transfer 


Paula Gubrud 


Nursing 


Professional/Technical 


Chris Heideman 


Occupational Therapy Assistant 


Professional/Technical 


Amy Hoover 


Aviation 


Professional/Technical 


Rebecca Kenney 


Continuing Education 


Other 


Vern Porter 


Welding 


Professional/Technical 


Jim Russell 


Business Administration 


Professional/Technical and Transfer 


Teri Tong 


Dental Hygiene 


Professional/Technical 


Valerie Ward 


Evening/Weekend College 


Other 



This report outlines the consultant’s findings from interviews with the pilot testers and consequent 
recommendations for implementing this assessment effort collegewide. The report is divided into 
findings about the process itself and findings about Assessment Builder, the software used during 
the pilot process to store and report the assessment information. 



The Educational Assessment Process 

The faculty and deans participating in the pilot test noted a variety of ways in which this 
institutional assessment process will be valuable for their programs and departments. For transfer 
and service programs that had not previously engaged in formal assessment activities, the process 
provided the opportunity to self-assess. Faculty from these programs found having to identify 
specific criteria and indicators on which they can be assessed helpful in thinking about ways to 
measure their past success and future improvement. They saw potential for gaining new 
information about their programs that could be used to guide future decision making and keep 
department members informed. 
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For programs with external accreditation requirements, the process provided a systematic method 
for tracking and storing program-related data needed for the accreditation process. In programs 
where the accreditation process has no prescribed structure, the institutional assessment process 
provided a framework for collecting and summarizing relevant information. Additionally, the pilot 
testers noted that any support provided as part of the institutional process will help them gain 
better access to this necessary information. 

For all programs, the pilot testers noted that the institutional assessment process provided a 
formal avenue for delineating needs and highlighting successes. Participants appreciated the 
opportunity to candidly describe their programs’ shortcomings and needs, to bring to the table 
arguments for increased resources, to defend their programs against potential cuts, and to use 
data to make strategic planning decisions. Because transfer programs are cross-disciplinary and 
accreditation of professional/technical programs may include evaluation of collegewide resources, 
several pilot testers also noted that this collegewide assessment effort fills an important role by 
addressing needs across all disciplines and areas. 

Recommendations for Implementation 

In a succinct question echoed by other participants, one pilot tester asked, “Do you want good 
data or do you just want me to fill in the blanks?” MHCC should seriously consider the effects of 
the implementation process on the quality and usefulness of the resulting data. If participating 
faculty and deans are to engage in this process meaningfully, it must be established in a way that is 
easy to implement and beneficial to the programs and disciplines as well as to the administration. 

To achieve this benefit, the process should be customized to each assessment unit (program, 
discipline, department, or division); include substantial assistance and support for implementation; 
include ongoing support and coordination of the process through the Research and Planning 
department; and close the feedback loop by ensuring that the assessment information is used by 
the administration, which in turn keeps the faculty informed. The following recommendations for 
effective collegewide implementation of the process are based on the information and ideas 
provided by the pilot testers. 
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Recommendation 1: Customize the indicators and process to be used to each 
program 

As part of the pilot process, the Assessment Task Force developed a list of standardized 
indicators associated with the college goals to be assessed by all departments and programs. With 
the understanding that different types of programs have different needs, slightly different 
indicators were chosen for professional/technical programs, transfer programs, and other 
programs (i.e., service departments such as Mathematics and nonacademic programs such as 
Continuing Education.) Exhibit 2 lists the indicators developed by the taskforce. 



RMC Research Corporation ♦Port land, OR 



4 



Exhibit 2 

Indicators Developed by the Assessment Taskforce 



College Goal: Teaching and Learning 


Prof ./Tech. 


Transfer 


Other 


Graduation/Completion 


X 






Job placement of graduates 


X 






Job placement of nongraduate core course completers 


X 






Jobs available related to skill training 


X 






Living wage 


X 






Number of transfers 




X 




Transfer performance 




X 




Course success 


X 


X 


X 


Sequential course success 




X 


X 


Student satisfaction — Course 


X 


X 


X 


Student satisfaction — Exit 


X 


X 




Student satisfaction — Follow up 


X 


X 




Student satisfaction — General ed assessment 




X 




Faculty and program satisfaction with college support services 


X 


X 


X 


Student, faculty, and program satisfaction with service received 






X 


College Goal: Enrollment 


Enrollment 


X 


X 


X 


Cost/Revenue per FTE 


X 


X 


X 


Unique to Portland area 


X 


X 


X 


Retention — Course 


X 


X 


X 


Retention — P rog ram 


X 






Retention — Sequential courses 




X 


X 


Marketing 


X 


X 


X 


College Goal: Technology 


Technology funding level 


X 


X 


X 


Equipment objective 


X 


X 


X 


College Goal: Facilities 


Facilities funding level 


X 


X 


X 


Maintenance schedule 


X 


X 


X 
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This list of indicators, however, did not prove to be sufficiently flexible. Virtually all pilot testers 
discovered that they could not meaningfully address some indicators listed for their program type. 
For instance, Dental Hygiene cannot assess “Retention — Course” because all students in the 
program take precisely the same courses and cannot drop courses without quitting the program. 

In Aviation, issues of job placement were hard to interpret because no student just finishing any 
college aviation program will have completed sufficient flight time to be directly hirable. The 
problem was far more acute for transfer programs whose student populations are not well defined, 
who rely on a variety of general education courses and courses from other departments, and 
whose facilities and equipment are often shared. (For instance, the BCT division shares facilities 
and technology between all departments.) And because “other” programs range from academic 
service departments like mathematics and English to noncredit services such as Adult Basic 
Education and Continuing Education, their needs are more diverse yet. 

Most pilot testers also found indicators missing from the list that are of specific importance to 
their programs or that they already use to measure success. For instance, Adult Basic Education 
needed to address indicators related to GED success and matriculation to regular courses, and the 
music department measures success by the standing of its performance groups in various 
competitions. 

Finally, some assessment units do not fall neatly into any of the three categories. Mental 
Health/Human Services, for instance, offers an AAS degree, but most of the program’s students 
transfer to baccalaureate programs. BCT offers both transfer and professional/technical programs, 
which are difficult to assess separately because they share courses and resources. To address these 
profound differences between different programs and departments, the following course of action 
is recommended. 

Common Indicators: From among the supportive and customized indicators outlined below, 
each program should choose the 3 to 6 common indicators that most significantly measure the 
success of the program. Common indicators are those that are critical to the viability of a program 
and will often be the same for similar types of programs; however, common indicators may be 
different for every assessment unit. For example, one common indicator for Continuing Education 
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may be “Student Satisfaction — Course” because each individual continuing education course is 
taken by choice rather than by program design. For a transfer program, a common indicator may 
be success of MHCC students after they transfer compared to the success of other students at the 
transfer institutions. 

Supportive Indicators: Using the list of indicators developed by the Assessment Taskforce as a 
starting point, each program should respond to any indicators on the list that are relevant and 
meaningful. 

Customized Indicators: Each program should develop and respond to any indicators in addition 
to the supportive indicators that it believes are good measures of its success. 

Unquestionably, accreditation teams and the MHCC Assessment Oversight Committee need to 
see consistent use of assessment across programs; however, it is neither practical nor worthwhile 
for every program to respond to every indicator listed for that program type. The necessary 
constancy can be achieved through use of a standardized process and reporting vehicle and 
on-going review of the same common indicators. 

Recommendation 2: Provide assistance and support for initial implementation 

For many faculty the process — and perhaps even the concept — of assessing program outcomes 
will be new, and neither the departments nor the Research and Planning office have data 
collection instruments in place. For other faculty accustomed to external accreditation, efficiently 
integrating this assessment process with other required processes will be challenging. Faculty in all 
programs and departments will need to be supported through the process of determining what 
defines programmatic success in relation to the college goals and how to obtain, record, and 
interpret data to measure related outcomes. 

Most of the interviewees indicated that during the pilot test they relied heavily on the use of a 
mentor, a Research and Planning staff person, or another person knowledgeable about the process 
and software. Most also indicated they would have liked more assistance. Some pilot testers 
needed help understanding the purpose and outcomes of the process, some needed help with the 
software itself, and most were confused about how to tie the process and the software together. 
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(Common questions included, “Where do I input this information? What goes where?”) Many 
testers also wished for feedback about whether they had completed the process correctly. 

Transfer and other non-professional/technical programs will need added help and support during 
the initial implementation of this process. Although most professional/technical programs already 
have standard assessment methods in place for specialized accreditation or Perkins grant 
evaluation reasons, transfer faculty and deans interviewed stressed that the process as piloted 
required them to “reinvent the wheel.” Transfer and other programs will need to see models or 
examples of completed assessment reports, will need assistance developing or locating assessment 
instruments, and will need help translating the data into usable information. And unlike 
professional/technical programs, whose goals tend to be uncomplicated and easily measured (e.g., 
students graduate from the program and place into related jobs), transfer programs will need help 
determining how to measure progress toward their more complex and sophisticated goals (e.g., 
students develop a broader and deeper understanding of relevant concepts). 

MHCC has already identified an outside consultant to serve as an ad hoc Assessment Coordinator 
during the first year of implementation of the process. This Assessment Coordinator should be 
instrumental in helping individual assessment units work through the process, from establishing 
measurable objectives to entering data into the software. The Assessment Coordinator will also 
serve as a liaison between the assessment units and the Research and Planning staff. Each 
assessment unit should also have a Unit Coordinator who will coordinate the effort for that unit 
and also serve as the liaison between the unit and the Assessment Coordinator. The following 
schedule for project implementation is recommended: 

September 2001 

During the fall inservice, all faculty and deans should be given an overview of the institutional 
assessment goals and the process to be used, be introduced to the Assessment Coordinator, and 
be given the opportunity to ask questions. 

The pilot testers consistently and uniformly misunderstood the role of the Assessment Builder 
software, equating entering information into the software with the process of institutional 
assessment. Because conducting meaningful program-level evaluation will require the participants 
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to separate the software from the process, the software should not be introduced during the 
inservice. Training on the software can be provided to the specific individuals who will use it at 
the time of need. 

Fall 2001 and Winter 2002 

During fall and winter terms of the 2001-2002 academic year, the Assessment Coordinator should 
meet with the Unit Coordinator and some or all faculty in each unit. The division dean should also 
attend this meeting. In some cases, it may be possible for all the programs in one division to meet 
at the same time. During the meeting the Assessment Coordinator should work with the dean and 
faculty to do the following: 

6. Discuss the purpose and background of this collegewide institutional assessment process 
and the specific role of program-level assessment. 

7. Discuss departmental goals and determine how to define successful achievement of those 
goals. 

8. Choose selected indicators and develop additional indicators as needed. Chose common 
indicators from among the list of indicators to be used. Create a written list of these 
indicators to serve as a starting point for data collection and reporting later in the year. 

9. Determine what instruments or other means will be used to assess all indicators selected 
and discuss what information these indicators will provide. Determine how and when the 
data will be gathered and processed and what support is needed from Research and 
Planning. 

Spring 2002 

In spring 2002 the Assessment Coordinator will meet with again with the Unit Coordinators to 
guide them through the process of organizing the information gathered and inputting it into the 
assessment software. During this meeting the Assessment Coordinator will offer assistance and 
suggest ways to present the information. 

Academic Year 2002-2003 

Limited support from either the Assessment Coordinator or a Research and Planning staff member 
will need to continue into the second year of implementation. Programs that participated during 
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the pilot test will be completing the formal process for the first time, and new questions will 
certainly emerge from programs completing the process for the second time. 

Recommendation 3: Provide institutional support for the Unit Coordinators 

The pilot testers repeatedly stressed that initial implementation of this process required a large 
time commitment. In single person departments, the full responsibility for completing the 
assessment fell on that one person. In larger departments, one person needed to solicit 
information and opinions and coordinate the efforts of various faculty members with conflicting 
priorities. (In the pilot test, limited time forced most participants in larger departments to simply 
complete the process alone.) 

As suggested in Recommendation 2, each assessment unit should have a Unit Coordinator. The 
administration should recognize and support the time required by this coordinator if the process is 
to produce meaningful, considered results. For the first year of implementation for each unit, Unit 
Coordinators should be compensated, monetarily or through release time, for the extra time 
required. 

A possible alternative would be to involve the department secretaries, other support staff, or a 
Research and Planning department staff member heavily in the process. These support persons 
would be responsible for gathering and compiling the information and entering it into the 
computer. This approach would lighten the workload of the Unit Coordinator, freeing up the 
coordinator’s time for interpreting the information and making recommendations. 

Recommendation 4: Provide ongoing support and coordination of the process 
through the Research and Planning department 

In addition to early hands-on support from the Assessment Coordinator, programs will need 
ongoing structured support from the Research and Planning department. Central to ability of the 
programs and departments to complete effective programmatic assessments is the need for 
current, accurate, and usable data. The role of Research and Planning should be to centralize data 
collection efforts, to process available data, and to be proactive in providing data to the programs. 
The role of the faculty and deans should simply be to assemble the data and determine what it 
means for their students and programs. To this end, the Research and Planning staff should put 
systems in place to carry out the following: 
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10. Work directly with the deans of programs with unrestricted student entry to establish a 
meaningful method for determining which students are enrolled. Subsequently, all data 
related to enrollment and retention should be based on these definitions. All pilot testers 
from transfer and other unrestricted entry programs expressed frustration with their 
inability to answer questions about enrollment and retention. 

1 1 . Provide timely, accurate, and useful data related to enrollment, retention, and course 
success. Data reports provided to the pilot testers were of limited use — for some 
programs virtually all entries were marked “N/A”. 

12. Provide a bank of assessment instruments related to student satisfaction (course, exit, 
follow-up, and general education) and the means to administer the instruments and 
process the resulting data. This may include assistance in locating and following up with 
former students. 

13. Locate and process other information as needed and requested by programs and work 
with faculty to develop instruments to serve a variety of needs. 

14. Respond to special needs of departments or programs related to assessment; for instance 
the nursing department needs to demonstrate validity and reliability of all instruments for 
accreditation. 

15. Provide assistance for transfer programs to determine how they compare with other 
colleges. 

In the long term, Research and Planning should continue to provide assistance and support for the 
assessment process and serve as a clearinghouse for assessment information and resources. With 
all departments and programs engaged in regular assessment activities, the demands on Research 
and Planning will be great. The department must be prepared and sufficiently staffed to address 
these increasing demands. 

Recommendation 5: Close the feedback loop 

As conceived by the Assessment Taskforce, the function of the institutional assessment effort is to 
create an opportunity for feedback between the programs and departments and the college 
administration. The creation of the Assessment Oversight Committee is an important part of this 
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process as it serves its function to “provide ongoing monitoring of the educational assessment 
process, results, and actions.” What remains unclear, however, is how the results and 
recommendations from the programs and departments will be transferred from the faculty and 
deans to the administration and how the administration will report back to the faculty. All 
recommendations resulting from this process that require resources or administrative approval 
need to follow a specific and consistent feedback loop as illustrated in Exhibit 3: 

16. The programs and disciplines make recommendations as part of the institutional 
assessment process. 

17. The deans submit overall division summaries to the President’s Council via the vice 
presidents and to the Director of Research and Planning who will work with the faculty 
chair to schedule review by the Assessment Oversight Committee. The division summaries 
report findings and recommendations from the programs and disciplines and indicate 
which items need to be addressed in the next planning/budgeting cycle. Through the 
division summaries, the deans are responsible for bringing forward any resource requests 
identified in program and discipline assessments. 

18. As part of its annual planning and budget development process, the President’s Council 
responds in writing to the planning objectives and resource requests made by the deans as 
reported in the division summaries. The responses are sent to the Assessment Oversight 
Committee and copied to the vice presidents, the deans, and the programs and disciplines. 

19. The Assessment Oversight Committee reviews the patterns of resource allocation and 
denial with special emphasis on the common indicators identified by each program or 
department. The committee submits an annual report to both the President’s Council and 
the MHCC District Board of Education. 

20. The Oversight Committee may initiate a comprehensive review of a program or discipline 
based on its findings from the annual review of assessment indicators or at the request of 
the President’s Council. A comprehensive review may take one of many forms, such as 
additional data collection, paneling area employers, or DACUM. Results from a 
comprehensive review are forwarded to both the President’s Council and the Oversight 
Committee, with the Oversight Committee submitting any recommendations to the 
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President’s Council. The President’s Council will issue its findings, recommendations and 
decisions in writing to the Assessment Oversight Committee and send copies to the vice 
presidents, the deans, and the relevant programs and disciplines. The Assessment 
Oversight Committee will include a summary of results and recommendations in its annual 
report to the MHCCD Board of Education. 



Exhibit 3 

The Assessment Reporting and Feedback Process 




The Assessment Builder Software 

Assessment Builder was the software used during the pilot process to store and report the 
assessment information. MHCC is beta testing this software as part of the pilot test. General 
reactions to the Assessment Builder software were mixed. A few pilot testers indicated that they 
found the software intuitive and logical with an easy to use, linear structure. More commonly, 
however, the pilot testers intensely disliked the software interface. (One tester called it “The worst 
software I’ve ever used.”) These users found the interface unfriendly, nonintuitive, and visually 
perplexing. They also noted that the software has a steep learning curve. 

Problems With the Assessment Builder Interface 

Specific problems with the Assessment Builder interface and suggested changes for the final 
version of the software are outlined below. 
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Navigation 

Some users found the navigation intuitive and were highly successful using the step-by-step 
instructions developed by one of the Assessment Taskforce members. Others, especially those 
who tend to think holistically rather than linearly, found the navigation exasperating. Several 
frustrated pilot testers commented that the software is not visually oriented and thus it is not 
possible to “feel your way around.” Specific concerns and suggestions for improvement include 
the following: 

2 1 . The most common and emphatic frustration with the software interface was the need to 
reenter the department and criterion information each time the user moves to a new 
indicator. (Because the software repeatedly returned to a screen listing the department as 
Dental Hygiene, one tester termed the experience “software Groundhog Day” in reference 
to the movie in which a man is forced to relive the same day repeatedly.) The software 
should be modified so that the user enters the department code only once and 
subsequently is able to move freely between indicators, a change that several testers felt 
would improve the software tremendously. 

22. Some testers found it difficult “to keep track of where you are in the software.” These 
users noted that paths do not branch logically, and the software offers no linear method to 
reverse navigate. After entering findings for an indicator, for instance, the user is sent back 
to an earlier screen. Some users could not find where they had entered certain information 
in order to modify it later, especially when they returned to the software at a later session. 
(One tester stated that some numbers she had entered simply disappeared.) One frustrated 
tester remarked that the software requires the user to “submit a little information in a lot 
of different places.” The navigation should be modified to require less movement between 
different screens or so that this movement is easier to follow. 

23. The format of the opening interface — many pages each containing many buttons — is 
daunting, especially to visual users. Further, the software does not allow the use of menus 
or other familiar navigation tools. Also, the user should be able to determine easily which 
areas of the form are for what general purpose. For instance, the opening interface could 
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be divided into three distinct sections: data input, data reporting, and administrative use 
(entering indicators and departments). 

Data Input 

The testers expressed several concerns related to how the data is input: 

24. A very small viewable space is provided to input potentially lengthy text responses. The 
testers were frequently frustrated because they could not see all of the text at once. The 
viewing space for text responses should be increased. 

25. Users were not able to keep other applications open at the same time as Assessment 
Builder and therefore could not cut and paste information from existing documents into 
the software. Because much of the information to be entered will come from other 
sources, it is essential that this cut and paste process be available. 

26. Users are required to enter their departments by discipline code, which most do not know, 
rather than by name. At a minimum, the list of departments should be presented 
alphabetically by department name rather than numerically by discipline code. 

27. Some users felt the input was too isolated from the output. These users did not understand 
how the information they input would translate to a meaningful document. A Print 
Preview option available on the input screen may help. 

28. Some form of online help is essential. The purpose of some buttons ( Special Information ) 
and the intended use of different fields ( Direct Measures and Indirect Measures versus 
Quantitative Results) are very unclear. 

Reporting 

Several users commented that much of the power of the software will come from its ability to 
generate graphs, a feature unavailable in the beta version. In the current version, the software only 
creates a single report, which many users found difficult to read. (One user noted that the report 
“takes pages and pages to say three things.”) Specific suggestions for improvement of the output 
follow: 
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29. The standard report should be reviewed and substantially improved for readability. For 
example, the indicator name should be highlighted, information about a single indicator 
should be prevented from spanning a page break, and items that are unimportant (such as 
the indicator code number) or left blank by the user should be suppressed. The report 
listing available indicators is similarly unreadable. 

30. Include a brief report summarizing the essential information for all indicators on 1 or 2 
pages. For instance, list only the name of the indicator, the achievement level, and the 
“recommendations”. 

31. Improve the report listing which indicators are available to respond to. Like the report of 
findings, this report is virtually unreadable. It is further confused because it includes a list 
of all indicators. 
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Technical Problems With the Software 

Additionally, the pilot testers commonly encountered three technical problems: 

32. The software crashed or froze up regularly. 

33. The software disappeared (or the users perceived that it disappeared) from individual 
workstations. At least two testers were unable to complete the work because they could 
not locate the software during the time they had set aside to work on the assessment 
process. 

34. The users could not print the reports and thus view their output from their work areas but 
had to request reports from Research and Planning. (One user wondered, “I enter the data 
then Dan sends it to me as a Word file — why not just type it as a Word file in the first 
place?”) 

Additional problems with the technical operation of the software included the following: 

35. The software is not dual-platform and some departments have only Macintosh computers. 

36. The username for entering the Assessment Builder software is the user’s first initial 
followed by last name. This order is opposite of the last name followed by first initial 
format used for e-mail and server access at MHCC. Remembering passwords, which can 
also differ from system passwords, was also an obstacle. 

Despite technical problems and dissatisfaction with the interface, many users appreciated the 
opportunity to record assessment-related data in a manner that is systematic, uniform, and 
consistent. Some pilot testers commented that the function of the software — storing and 
organizing assessment data and noting departmental expectations and problems — is effective and 
useful. A few users, however, insisted that the software is “not worth what you get out of it”, and 
would prefer simply to enter the information into Word or Excel. 
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General Recommendations 

For any software to be used effectively by large number of faculty and staff members collegewide, 
the interface must be intuitive and user friendly. As one tester noted, the software “is just a way to 
organize the information — it should not be an obstacle.” Consequently, one of the following three 
courses of action is recommended: 

37. Use the software only for data input by a few trained users. Departmental faculty and staff 
could submit information, perhaps using a Word template, to the Research and Planning 
staff who would then coordinate the data entry into Assessment Builder. Alternately, one 
Unit Coordinator in each division could be individually trained in the use and 
troubleshooting of the software and could provide one-on-one assistance to the other Unit 
Coordinators. 

38. Engage a consultant to develop a custom database for the specific assessment needs of 
MHCC rather than using the packaged Assessment Builder software. 

39. Request that the software publisher, for whom MHCC is beta testing the product, 
overhaul the interface as outlined. If the publisher is unable to make the changes in a 
timely manner, it may be necessary to revert to the first recommendation (using the 
software only for data input) in the interim. If this course of action is chosen, the primary 
technical problems outlined above must also be addressed before using the software 
collegewide. 




RMC Research Corporation ♦Portland, OR 



51 



18 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 



X 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



q EFF-089 (1/2003) 

ERIC 



